Understanding Complex Energy Landscapes and Rare Events

Introduction T he amount of data in our world has been increasing at a rapid rate in a wide variety of applied disciplines such as in fi nance (the advent of high-frequency algorithmic trading) and genetics (recent experimental advances have led to vast amounts of new data-sets). As a result, analyzing large data sets is becoming a key basis of competition, supporting new waves of productivity growth and technological innovations. The primary role of many statisticians has evolved to develop advanced and often highly complex stochastic models to determine the underlying physical phenomenon in the datasets. In many of these statistical models, it is important to estimate a variety of parameters which can accurately describe the data. For example, in the fi nancial industry, fund managers are concerned with the " price volatility " of the fi nancial assets as it helps them allocate their investments to balance the risk level. In such scenarios, this information is obtained by evaluating an integral, or a " theoretical average " , in very high-dimensional space. If a fi nancial data set consists of only a time-series of prices, the integration task has to be carried out in as much as over 1000 dimensions to arrive at a correct estimate of the price volatility In practice to calculate this integral, the analytical value is not known due to the high-dimensional nature of the integral (i.e. one cannot simply perform mathematical calculations on a computer/by hand and obtain the answer). As a result, this has lead to a substantial literature on numerical approximation of integrals, which can be roughly divided into two main groups-stochastic methods and deterministic methods. It is widely believed that stochastic numerical integration methods out-perform deterministic methods for dimensions greater than three. In general, statisticians and probabilists focus upon stochastic ideas, in particular Monte Carlo approaches. Monte Carlo methods are based on the use of random numbers and probability statistics to investigate problems, and are believed (wrongly) to always break the curse of dimensionality and accurately estimate integrals in very high-dimensions. This is true in some scenarios, but there are instances whereby the methodology has very poor performance and yields inaccurate outcomes. That is, if the dimension of the problem is d, then it is thought that the method can work well if, for some " good " behavior the cost is polynomial function of d, and work poorly if the …